ref(consumer): remove the BLQ (backlog-queue) mechanism - #8166
Merged
Conversation
The backlog-queue router (BLQRouter) redirected stale messages, by Kafka timestamp, to the DLQ topic and panicked the consumer to flush in-memory state on the fresh->stale transition. It was never fully rolled out, the FSM (Idle/RoutingStale/Flushing/Forwarding + panic-to-reset) was hard to reason about, and no storage enabled it in prod. Remove the strategy, its wiring in factory_v2/consumer, the bench fields, and the consumer.blq_* sentry-options. The DLQ policy/topic are untouched. This clears the way for a simpler DLQ-by-age lever built on the standard InvalidMessage mechanism. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
phacops
approved these changes
Jul 14, 2026
onewland
added a commit
that referenced
this pull request
Jul 17, 2026
> **Stacked on #8166** (BLQ removal). Base is `ref/remove-blq`; review that first. Retarget to `master` once #8166 merges. ## Summary Adds `DlqByAge`, a stateless arroyo strategy for the Rust consumer that sheds load during a backlog. When enabled, it routes a **configurable proportion** of messages older than a threshold (by **Kafka broker timestamp**) to the DLQ via the standard `InvalidMessage` mechanism. The rest are forwarded untouched — so a backlogged consumer catches up to fresh data while the parked backlog is backfilled from the DLQ later. This replaces the removed BLQ mechanism with something far simpler: **no state machine, no consumer panic, no separate producer.** It leans entirely on arroyo's existing DLQ buffer + policy. ## How it works - Sits at the front of the pipeline (where `BLQRouter` was), *before* decode/process, so dropped messages cost almost nothing. - For each message: if the per-storage sample rate is 0 → forward (fast path). Else if `now - broker_ts <= threshold` → forward. Else roll `rand < rate`: hit → `Err(SubmitError::InvalidMessage)` (→ DLQ); miss → forward. - **Only inserted when a DLQ topic is configured** (`dlq_configured` flag from `consumer.rs`). Without a DLQ policy, arroyo logs an `InvalidMessage` and silently drops it — which would lose data — so we never wire the strategy in that case. ## Kept out of Sentry Age-based routing is an expected load-shedding outcome, not an error. Dead-lettered messages use `InvalidMessageReason::Ignored`, so arroyo logs them at DEBUG (not ERROR) and the tracing→Sentry layer (`logging.rs`) does **not** turn them into Sentry issues. Both reasons produce to the DLQ identically — only the log level differs. This is the same intent as the `SilencedDLQMessage` path from #8012. ## Configuration (sentry-options, read per-message, runtime-tunable) | Option | Type | Default | Meaning | |---|---|---|---| | `consumer.dlq_by_age_sample_rate_by_storage` | dict `{storage: number}` | `{}` | fraction (0–1) of too-old messages to dead-letter; no entry ⇒ 0 (disabled) | | `consumer.dlq_by_age_threshold_seconds` | int | `3600` | age cutoff by broker timestamp | **Ships disabled** — no storage is enabled and the rate defaults to 0, so this is an inert passthrough until an operator opts a storage in. Metrics: `dlq_by_age.stale_seen` and `dlq_by_age.routed_to_dlq` (storage is already a global metric tag). ## Test plan - `cargo test --lib dlq_by_age` — 6 unit tests pass: disabled-by-default, explicit rate 0, fresh-forwarded, rate 1 routes all stale, per-storage scoping, and a partial-rate split. Each DLQ'd message is asserted to carry `Ignored`. - `cargo clippy --all-targets` clean; `cargo fmt --check` clean. ## Follow-up (PR 3, ops) Enable per storage via sentry-options and watch `dlq_by_age.routed_to_dlq` + DLQ topic lag. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the backlog-queue (BLQ) mechanism from the Rust consumer.
BLQRouterredirected stale messages (by Kafka broker timestamp) to the DLQ topic and panicked the consumer to flush in-memory state on the fresh→stale transition. It was never fully rolled out, the FSM (Idle/RoutingStale/Flushing/Forwarding+ panic-to-reset) was hard to reason about, and no storage enabled it in prod.This is a pure removal (−481 lines, no additions). The DLQ policy/topic are untouched.
Removed
rust_snuba/src/strategies/blq_router.rs(the strategy + its unit tests)pub mod blq_router;instrategies/mod.rsblq_producer_config/blq_topicfields, and the pipeline-wrapping block infactory_v2.rsblq_producer_configbuilder + factory args inconsumer.rsblq_*: Nonebench fieldsconsumer.blq_*options from the sentry-options schemaWhy
Clears the way for a simpler DLQ-by-age lever (follow-up PR) built on the standard
InvalidMessagemechanism, so we can shed a configurable proportion of too-old messages to the DLQ during a backlog and backfill later.Test plan
cargo check --all-targets— clean, no warningscargo test --lib strategies— 36 pass🤖 Generated with Claude Code